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ABSTRACT 

The presentction encompasses a conceptualization, of 
what is meant by evaluation and what is involved in evaluating 
training programs which pre] -are special educators* Specifically, 
evaluation is differentiates from research; Stake's general framework 
for evaluating educational programs is summarized; and a number of 
factors involved in conceptualizing the nature and scope of training 
program evaluation are considered, as are several critical problems 
related to such evaluation. Finally, some specific ideas are offered 
with reference to the evalurtion of special education training 
programs. (A 7-item bibliography is included*) (Author) 
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Until recently, the question of hov; to evaluate, systematically 
and comprehensively, the nature and worth of preparation programs for 
O the education professions generally was ignored. Currently, it is one 

of the most discussed and least understood concerns in the field of 

o 

education. This paper encompasses an attempJ- to present a brief concep- 
tual framework for understanding what is meant by the term evaluation 
and what is involved in evaluating programs which prepare special educa- 
tors. 

Evaluation and R es earch Differentiate d 

\ For purposes of the following discussion, evaluation is defined 

' , • 

y as that process by which attempts arc made Lo understand Evograms lu 

order to describe, predict, explain, and make decisiom?, e.g., determining 
the over-all impact and value of a training program or of an individual 
professional's program. (By way of contrast, iLSsessmc;nt is defined as 
that process by which an individual attempts to understand himself and 
' other individua :.s in order to describe, predict, explain, .".nd make 
^ ' decisions, e.g., a professional assessing himself or an exceptional 

child.) Stake and Denny (1969) have expressed the goal of evaluation 

*This paper is based on ideas presented in a monograph entitled 
"Teachc-v education and the educationally handicapped: Seme basic issues 
Srtd some pa-rLial answers" which was prepared in connectioi with Elemen- 
tary and Secondary Education Act (ESEA) Title VI-B, Project No. S-006-0000- 
768/002. Cah.fornia State Department of Education, Division of Special 
Education, July, 1970. 
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as follows: "Evaluation is not a search for cause and effect, an 
inventory of present status, or a prediction of future success. It is 
something of all of these but only as they contribute to understanding 
substance, function, and worth/* 

Most writers in this area have made a distinction between evaluation 
and research as related to educational programs, and the distinction has 
been conceptualized in a number of ways. In its most basic form, eval- 
uation may be viev;ed as any process by which information is gathered 
and judgments are made about a specific program. Often such information 
is non-generalizable because of the lack of appropriate standards by 
which appropriate relative and/or absolute comparisions might be made. In 
contrast, educational research which focuses on program evaluation may be 
viewed as a process by which information is systematically gathered using 
carefully controlled procedures and appropriate comparisons, thereby 
producing information which may have widespread implications. Mclntyre, 
Meierhenry, Hoffman, Baldwin, and Fredericks (1969) distinguish between 
evaluation and research as related to education programs by conceptualizing 
the two as being on a continuum with informal evaluations at one end and 
highly controlled comprehensive research efforts at the other end. 

Perhaps the greatest value of the distinction bet-zeen program eval- 
uation and research is not so much that it clarifies the conceptual 
difference betvjeen the tv;o but that it clarifies the limitations of 
many current evaluative efforts. Ideally, all programs .should be 
comprehensively evaluated using a research design v/hich allows for 
absolute and/or relative comparisons with appropriate standards. Such 
formal and systemr'.ic evaluations would provide both useful feedback 
for a specific program and gencralizable information which v;ould be of 
value to others, e,g,, the data collected could make a substantial contri- 
bution .to efforts to deal with basic issues confronting \he field of education 
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Stake's General Framework for Evaluating Educational Programs 

In conceptualizing the various facets which should be considered in 
attempts to evaluate current training efforts, it is helpful to begin with 
the general conceptual framework for evaluating educational programs v/hich 
has been formulated by Robert Stake (1967).- In brief. Stake emphasizes 
that "the two basic acts of evaluation" are description and judgment, and 
both are needed if programs are to be understood (see Figure 1). In 
addition, his conceptualization clarifies that, if a program is to be 
fully described and judged, there must be data (a) for evaluating the 
functional contingencies between antecedent conditions, transactions, 
and outcomes, (b) for evaluating the congruence, betvzeen what is intended 
and what occurs, and (c) for making absolute comparisons (based on standards 
of excellence) and/or relative comparisons. Obviously, such a matrix 
of data would provide much of the informaton needed for describing, 
demonstrating the effectiveness of, and improving a program's basic 
propositions and goals, content and process, as well as for making 
general decisions about such programs. 
Key Factors in Evaluating; Training, Programs 

A number of factors should be considered in conceptualizing the 
nature and scope of training program evaluation. First, it is clear 

*In addition to Stake's article, other literature resources with 
which the concerned reader might want to become familiar are: (a) the 
series of reports published by the UCLA Centers for the Study of Eval- 
uation and for the Study of Evaluation on Instructional Programs; (b) 
the discussion of the National Assessment of Educational Progress pre- 
sented in Caps Capsu le (1970); (c) the process model formulated by Kaufman 
(1971); (d) the major volume on educational evaluation edited by Tyler 
(1969); and (c) the April 1970, issue of the Review of Educational 
Research v/hich w > devoted to educational evaluation. 
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A layout of statements and data to be collected by the evaluator of an 
educational program. 



Fig. 1. Stake's graphic representation of his conceptual framework 
for program evaluation 
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that Stake's framework has direct application in efforts to evaluate 
programs which prepare education professionals. Such evaluation, however, 
encompasses the direct application of the framework not only to such a 
program, but also to specific district and school programs in which the 
training program's staff, participants, and graduates are involved. For 
example, iu addition to investigating the impact on the trainees (such as 
their ability to plan and implement a special lesson), data should be 
gathered on the pupils with whom they v;ork (such as whether the pupils 
learn the skills included in the lesson) and on the effect the program's 
graduates have on the districts and schools in which they are employed 
(such as whether they stimulate changes in basic policies regarding methods 
and materials) . 

Second, in evaluating any educational program, it is important to 
determine not only the congruence between \/hat is intended and v/hat occurs, 
but also to investigate possible major side effects. For example, most 
programs do not lave well delineated objectives in the affective domain, 
and therefore, data often is not collected regarding the program's impact 
in this area. This is unfortunate since two programs which produce 
professionals of equal ability with reference to stated performance criteria 
may produce individuals with very different attitudes regarding the field 
of Education. 

Another critical variable to be considered is the time at which the 
evaluation is carried out. It is evident that all formal educational 
programs are lengthy and that educational programming is most appropriately 
patterned and sequenced with reference to long range goals rather than 
immediate instructional objectives. Indeed, the most relevant criterion 



for evaluating a program's success is the long range impact, and thus it 
should be recognized that the use of immediate objectives as criteria may 
be misleading. For example, the positive or negative impact of something 
learned today may only be reflected at a later time; furthermore, the fact 
that something is not learne^' at a particular moment is not tantamount to 
saying that it should have been learned at that moment, for it well may 
be that it will be more easily mastered at a subsequent time* Thus, in 
view of such temporal factors, it is evident that the differences between 
two groups of individuals from different training programs may not be 
apparent at the conclusion of their respective programs but may be very 
evident two years later. 

Further complications arise from the impact of individual difference 
variables • For example, a procedure may prove to be more effective for an 
individual with a certain pattern of personality characteristics than for 
a person i;ith a different pattern. 

And, of course, it is necessary to consider the amount of economic 
support (time, staff, space, etc.) required to bring about particular effects 
For example, the accomplishments of a new procedure must be evaluated v;ith 
reference to cost factors in order to determine its feasibility for large 
scale implementation. 

Finally, since aJl educational progra.i\s need to be improved, a comp- 
rehensive evaluation of a training program requires an investigation of 
the degree to which evaluative feedback is used systematically to improve 
various aspects of the program, e,g,, content: and process. 
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Critical Problems Related to Evaluating Train ing P rograms 

As the preceding discussion suggests, comprehensive program evaluation 
is complex. In addition to this complexity, there are some serious problems 
which must be overcome before the comprehensive evaluation of training 
programs in special education can be accomplished. 

Besides the very real practical problems related to attitudes toward 
and the financial costs of comprehensive program evaluation, there are a 
number of problems related to v;hat should be measured and how to measure 
it. One of these critical problems stems from t:he failure of educators to 
specify competencies (knov;ledgc, skills, and attitudes) v/hich are to be 
developed by the training program. Without clearly stated behavioral 
objectives, those responsible for evaluating the program arc seriously 
handicapped in their efforts (a) to establish appropriate priorities 
regarding what is to be investigated and what the performance indicators arc 
to he, (b) to evaluate (sample) the congruence between what is intended 
and what occurs, (c) to investigate possible side effe.cts, aid so forth. 

Another critical problem is that appropriate measures and procedures 
for evaluating some very important aspects of training programs have not 
been developed. And the reason for this state of affairs is not simply 
the absence of the knov/ledgc and skill needed to develop them. (It seems 
reasonable to suggest that many program evaluators and developers of 
measures and procedures used in evaluative investigations tend to limit 
their efforts to those areas which our society values and rewards.) 
Whatever the reason, however, the lack of availability of appropriate 
measures and procedures has made it impossible, to date, to even contemplate 
fully evaluating an educational program. 
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The- resolution of the above problems will require considerable time 

ard resources, and in the meantime, program evaliaticn will suffer from 

a variety of inadequacies. This fact gives rise to another problem, 

i.e., a reaction against program evaluation. There are many individuals 

and groups who v;ould prefer to sec no evaluation rather than take a chance 

that a program will be evaluated in an inadequate (unreliable and/or 

invalid) fashion. These critics point to those instances when evaluative 

procedures and data have been misused and abused. For example, some special 

educators point to the tendency (e.g., on the part of legislators) to have 

special education programs evaluated primarily in terms of immediate 

achievement benefits to children and cost accounting procedures. (There 

has been a trend to judge a t.raining program's benefits in terms of 

immediately measureable improv<Mnent in the "3 R's" among the children 

served by the personnel trained in that program; moreover? it has been 

> 

suggested that the amount of improvement should be judged with reference 
to whether it warrants the fiscal expenditure per trainee and per pupil. 
On the surface, such criteria may appear to be reasonable. However, in 
light of our current limited kaov;ledge regarding effective strategies 
for educating the majority of exceptional children, this level of 
evaluation is probably premature and is certainly not comprehensive 
enough.) Clearly, the use of such inappropriate evaluative criteria 
is lamentable. Equally lamentable, howevei, is the tend0.ncy to suggest 
that such misuses of the evaluative process justify the continuing 
absence of formal evaluation v;hich characterize^, so many special 
education programs. The misuses and abuses of the evaluative process 
do not invalidate the importance and usefulness of evaluation. Indeed, 



it should be emphasized that much of the criticism which has been directed 
at the inadequacy of current procedures, "and the unfairness of decisions 
based on them, represents a localizing in the tool of the blame for the 
lack of clarity which characterizes the thinking of citizens of this 
democratic society, for it is the citizenry who determine the values and 
policies which direct the use of society's technical methods" (Adelman, 
Zimmerman, and Sperber, 1969) • Thus, the reaction against program 
evaluation is viev/cd as inappropriate; this, of course, does not make 
the problem any less real. 

Obviously other examples could be offered of problems which confront 
program evaluators, Hov;ever, it is felt that the problems which have been 
discussed are, currently, the mjijor deterrents to the comprehensive 
avail ation of training programs in special (and general) education. 

Some Tiou ghts on E valuating Special Education Training, Pr ograms 

Within the limitations set by such problems as those which have been 
described above, any program should attempt to evaluate as wide a range of 
impact as possible using procedures and standards which allow for objective 
and generalizable conclusions. For example, a comprehensive evaluation 
might encompass an investigation of the program's impact on (1) the partici- 
pants, (2) the pupils who arc served directly and indirectly as a result of 
the efforts of the program's participants and graduates, and (3) the field in 
general. The primary emphasis in such an evaluation should be on describing 
and judging the congruence betv;een stated instructional objectives and what 
is accomplished, but there also should be an investigation of possible major 
(positive and negative) side effects. 
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To be more specific about the nature and scope of such evaluative 
efforts, an investigation of the program^s impact might focus on^ 

(1) the participants v/ith particular reference to (a) the acquisition 
of new competencies (knowledge, skills, and attitudes), e.g., new teaching 
procedures, and (b) the modification of existing competencies, e.g., accept- 
ance of personal responsibility for learning needed competencies, involve- 
ment in the field; 

(2) the pupils v;hoin they serve v;ith particular reference to (a) the 
remediation of underlying process deficits, interfering behaviors, or 
both, e.g., perceptual deficits, extreme withdrawal and passivity, (b) the 
acquisition of needed pre-requisites , e.g., attending, listening, (c) achieve- 
ment in basic school subjects, e.g., reading, language, mathematics, and 

(d) relevant other behaviors and attitudes, e,g., self-direction, self- 
evaluation, inter-student cooperation, feelings toward school; 

(3) the field with particular reference to (a) the number of 
professionals , para-prof essionais , and recruits who are influenced directly 
and indirectly, (b) effects on spec.ific school districts which probably 
would not have occurred if the program did not exist, e.g., changes in 
policies and practices related to classroom methods and materials, staffing, 
in-service training, and so forth which were facilitated by the program^s 
staff, student^, and/or graduates, (c) effects on specific institutions of 
higher education, e.g., changes in policies and practices related to pre- 
service training, and (d) effects on educational thought in general, e.g., 
changes in conceptualization regarding the purposes and processes of formal 
education. 
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Pertinent data can be gathered by employing rating scales, question- 
naires, objectives tests, essays, Q sorts, systematic analyses of perform- 
ance, systematic records of specific accomplishments, directly solicited 
evaluations, measures of such concepts as anxiety, locus of control, inde- 
pendence and self control, expectations and aspirations, and so forth, 
(Obviously, whenever possible, standardized procedures should be used,) 
The sources for such data range from individuals involved in the program to 
representatives of a variety of institutions; the most likely sources are 
the training program staff, the program participants, qualified individuals 
who are not affiliated with the program (who v7ill be impartial), various 
school personnel (administrators, supervisors, teachers), pupils, parents, 
and subsequent employers and colleagues. 

As the examples offered in this section suggest, programs which prepare 
special educators can and should be evaluated on many levels. In addition, 
it should be evident that the concerns, issues, and problems related to 
evaluating training programs in both general and special education are not 
substantitvely different and that the process of evaluating such programs 
is in its early developmental stages. 

Concluding Statement 

Until there is a more definitive body of knowledge in the field of 
special education and further development with reference ':o the processes 
by vjhich v;e train professionals and evaluate such training, it seems 
unlikely that training programs for special educators can be evaluated 
satisfactorily. Nevertheless, such programs must and should be evaluated, 
and those responsible for the programs should be held accountable. However, 
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the t-crm accountability must not be interpreted simplistically • At this 
time, appropriate program evaluation in special e :equires more 

than the systematic collection of immediate achievement and cost accounting 
data, in particular, it is felt that programs V7hich prepare special 
educators should be evaluated comprehensively in terms of their general 
contribution to current educational services, training, and research, 
rather than in terms of such narrow criteria as pupil achievement in the 
"3 R's" or per capita cost with reference to immediate pupil benefits • 
Clearly, there is a great deal which still must be learned 'about educating 
exceptional individuals, training professionals, and evaluating educational 
programs; we cannot; afford to ignore the implications of these needs in 
the rush to establish strategies for accountability. 
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